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Nonlinear programming methods are used to solve saddle-point prob- 
lems subject to inequality constraints on the variables; in particular, the 
type of saddle-point problem arising in pursuit-evasion differential games 
is considered. The methods investigated fall into two groups: solution 
of the nonlinear simultaneous equations obtained from the Kuhn-Tucker 
conditions, and solution of a sequence of constrained optimization prob- 
lems by the gradient projection algorithm. These methods are applicable 
to any real-valued function f(x,y) which is convex in x, concave in y_, and 
has continuous and bounded second partial derivatives. Several examples 
are given which illustrate the characteristics of the numerical procedures . 
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I. INTRODUCTION 



The original development of differential game theory is credited to 
Isaacs for his work during the 1950's [8] . His book [9] , published in 
1965, provides a comprehensive summary of this theory. Issacs' develop- 
ment closely resembles the dynamic programming approach to optimal 
control theory popularized by Bellman [2] . Many subsequent authors have 
studied differential games; among the significant contributors are Ho, 
Bryson, and Baron [6] , and Berkowitz [3]. These and many recent papers, 
use the calculus of variations as an analytical tool. 

A statement of the differential game problem is as follows: A dyna- 
mic system is described by the state equation 

x = f_(x(t) , u(t) , v (t) , t) ; x( t o) = 2io (1) 

where x is a vector which describes the position, or state, of the game, 
u is a control vector selected by player 1 and v is a control vector selec- 
ted by player 2 . The vectors _u and v are members of the sets U and V 
respectively, denoted by uf U and v e V, where U and V are to be speci- 
fied later. The performance criterion is a functional assumed to be of the 
form 



J = h(x(t, ) , t f ) + 



f g(x(t), u(t), v(t), t)dt , 
J t 

0 



( 2 ) 



which player 1 wishes to maximize, and player 2 wishes to minimize. A 
saddle point of J is sought (if it exists), i.e. , a u*e U and v*cV are 
sought such that 
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J(u,v*) ^ J (u* , v*) ^ J(u*, v) 



(3) 



In the terminology of game theory, J is the payoff, u and v_ are strategies 
of the two opposing players, and _u* and v* are optimal pure strategies. 

A strategy is a decision rule that specifies a control to be applied by a 
player, based on all information available to him, in any given situation. 
Generally, the literature has considered a strategy to be a feedback con- 
trol law, that is , u and v are specified in terms of the states x; however, 
Willman [20] has used the concept of an open-loop strategy. In terms of 
the previous definition, this means that the information available to each 
player is restricted to the initial state and time, and the terminal time. 
Note that u* and v* are minimax strategies in the sense that player 1 
maximizes his minimum gain while player 2 minimizes his maximum loss. 

The concept of fixed terminal time ignores the fact that in more 
general games the game may not terminate at all. Berkowitz [4] requires 
a differential game to terminate whenever t, and x(t) are such that 
(t,x(t)) is a point of a previously specified set in (t,x) space. This is 
the "terminal surface" discussed by Isaacs [9] . Berkowitz [4] also 
restricts the term "differential game" to include only two-person, zero- 
sum games (the type described above), although others [19] have not 
been so restrictive and have extended the theory to N-player, non-zero- 
sum games. Zero=sum refers to the fact that there is a single performance 
criterion which one player tries to maximize while the other tries to min- 
imize. Thus, the maximizing player's gain is the minimizing player's loss 
and the sum of this gain and loss is zero. In a non-zero-sum game, the 

objectives of the two players are not directly opposed -- each may have 
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an individual performance measure to minimize , and the sum of the two 
players' criteria is not necessarily zero. It may also be that there are 
more than two players, each controlling an input to a single system, and 
each trying to minimize his individual performance criterion. Such prob- 
lems are not considered in this thesis. 

Although variational techniques seem to be an appropriate method 
for solving differential games, a distinct drawback is that inequality 
constraints on the value of the state and control variables at any time 
are difficult to include in the analysis [4] . Thus, more tractable methods 
are sought. Since the problem is basically an optimization problem sub- 
ject to inequality constraints it is natural to consider mathematical pro- 
gramming techniques (nonlinear programming in particular since the 
performance measure will generally be nonlinear). As far as this author 
can determine, only one other paper [10] has considered this approach. 

The nonlinear programming problem was first stated mathematically 
by Kuhn and Tucker [14] . The problem is to find an x° that maximizes a 
function F(x) subject to inequality constraints of the form Xl^jSO 1 and x 
^0_. Kuhn and Tucker found necessary conditions for x° to be a solution 
to this problem. If F(x) is a concave function in the region where 
X(x)so. and x ^0^, the conditions are also sufficient. Several algorithms 
to solve this problem have been proposed although this thesis will use 
only Rosen's gradient projection algorithm [15] . 



1 X(x)-0^ means that each component of the vector X(^) is non- 



negative . 



9 



The use of nonlinear programming methods to solve optimal control 
problems has been discussed by Rosen [16] , [17] , and by Kirk [12] . 

This thesis will discuss the connection between nonlinear programming 
and saddle-point problems such as arise in pursuit-evasion differential 
games, and several examples which illustrate these ideas will be solved. 
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II. BASIC PRINCIPLES 



Several results from Game Theory will be required and are stated 
here. A more complete discussion of these theorems and their proofs can 
be found in [ll] . 

Theorem 1_. General Min-Max Theorem. Let f(x, y) be a real-valued 

function of two vectors x and y_ which lie in X and Y, respectively, where 

both X and Y are closed, bounded, convex sets. 3 If f is continuous, 

convex in x for each y, and concave in y_ for each x, 3 then 

min max f(x,y) = max ,,min f(x,y) 
x€ X Yf Y Y € Y x^ x 

A sufficient condition for a saddle point follows from the next 



theorem . 

Theorem 2 . A necessary and sufficient condition for 

min v max f(x,y) = max min f(x,v) . 

X€ X ycY — ^ ye Y xe X — ^ 

is that there exist a pair x*€ X, y*e Y such that for all y_ in Y and all 

x in X 



2 X is a convex set if the straight line joining any two points 
in X lies entirely within X. That is , if x 1 , x 3 e X then 

x 3 = 6x 1 + (l-0)x 3 c X for all 0^ 0 ^ 1 

3 f(x) is a convex function if 

(l-6)f(x 0 ) + 6f(x 1 ) * f([l-e]x° + fix 1 ) W 

for 0 ^ 6 £ 1 and for all x 0 and x 1 . The negative of a convex 
function is a concave function, i.e., f(x) is a concave function if the 
sign of the inequality in (i) is reversed. 
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f(x*,x) *f(x* ( £*) ^f(x,£*) 



(4) 



Thus, a sufficient condition for the existence of a saddle point (in 
the sense of (4) above) is that f(x, _y) be convex in x for all y and concave 
in y_ for all x. 

Considering each side of the double inequality in (4) separately, 
the problem can be viewed as two optimization problems . Looking at the 
right-hand inequality first, and specifying the set Y as Y = (y_l X (y) ^ 0 , 
i=l , the problem 

where y is an m-vector, is a statement of a Maximum Problem from non- 
linear programming. If f(x,y) has continuous first partial derivatives, 
the Kuhn-Tucker necessary conditions can be written. 4 For y* to be a 
solution to the above problem it is necessary that y* and the vector of 
Lagrange multipliers ju ^ satisfy the following conditions 
l 



+ t U . 7 X , (y*)] 
i=1 yi x. yi 


= 0 


(5) 


X (y*) T ct r 


= 0 


(6) 


-y *=y 






X (y*) 


s o 


(7) 




s 0 


(8) 



4 The original problem considered by Kuhn and Tucker also included 
a nonnegativity constraint on the variables, y£0 > but general usage 
has associated the term Kuhn-Tucker conditions with the necessary con- 
ditions for this problem as well as the one they originally considered. 



where 7^ f(x*, y_*) denotes the gradient of f(x*, y) with respect to y_ 
evaluated at x*, y * , i .e . , 



7 Z f (2S*- Y*) 




Y = Y* 



L Sy m 

and that the constraint qualification [14] be satisfied. The constraint 
qualification rules out pathological behavior on the boundary of the con- 
straint set. In this thesis, only linear constraints will be considered, 
hence the constraint qualification is always satisfied. It has been shown 
[14] that if f(x*, y) is a concave function of y_, then the conditions (5)- 
(8) are also sufficient. 

Similarly, for the problem. 



mi n XfX f (£, X*) 



subject to X . (x) s 0 
xi — 



i=l , 2 , . . . ,k 



with f a convex differentiable function of x , an n-vector, the necessary 
and sufficient conditions are, 

k 



i=l 



XI X XI 



“X 



-x 



“X 



— X 



= 0 


(9) 


= 0 


(10) 


s £ 


(ID 


IV 

|o 


(12) 


s . Thus , 


for (x* , y*) to be 



solution of the constrained saddle-point problem, equations (5), (6), (9), 
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and (10) and inequalities (7), (8) , (11), and (12) must be satisfied. 
Although the Kuhn-Tucker conditions require only that the function be 
real-valued, concave-convex, and differentiable to ensure that a saddle 
point exists, the methods examined in this thesis will also require the 
function to have continuous and bounded second partial derivatives . 
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III. NUMERICAL METHODS 



The methods investigated for solving the constrained saddle-point 
problem may be divided into two groups; 1) solution of the nonlinear 
simultaneous equations given by the Kuhn-Tucker conditions , and 2) 
direct solution of a maximization problem and a minimization problem. 
Methods in the first group have the disadvantage that although the original 
problem is of dimension m+n, the Langrange multipliers must also be 
treated as variables and the dimensionality increases to m+n+k+1 . 

A. SOLUTION OF NONLINEAR SIMULTANEOUS EQUATIONS 

1 . Newton-Raphson Method 

Jacob and Polak [10] suggest a generalized Newton-Raphson 
method for solving the system of four(vector) equations (5), (6), (9), and 
(10) subject to the four(vector) inequalities (7), (8), (11), and (12). Con- 
vergence of this method was found to be somewhat sensitive to the 
starting point, although convergence was quite rapid if appropriate start- 
ing points were chosen. 

2 . Brown's Algorithm 5 

Brown [5] has proposed an algorithm to solve nonlinear simul- 
taneous equations which requires fewer multiplications than the Newton- 
Raphson method. It is quite similar to the Gauss-Seidel process for 
nonlinear systems of equations. Brown has observed that the stability 

5 Subroutine NLNSYS at NPS Computer Facility. 
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and convergence cf the algorithm do not seem to be dependent on the 
exact evaluation of the partials and first differences may be used to 
approximate the partials. This method was also found to be sensitive to 
the starting point. 

3 . G radient Pr ojection D etermination o f the Roots of the Equations 
given by the N ece s s ary Conditions 6 

This method suggested itself due to the nonnegativity con- 
straints on the multipliers in the necessary conditions. There are several 
ways of formulating this problem; the one chosen was to minimize 

J = C X x (x) T ^3® + [My) T ^ 3 2 (13) 

subject to 



-7 f(x,y) + S ju . V X .(x) = 0 

X — " . XI X XI — — 

— 1=1 — 

i 

7 f{x,Y) + r M • 7 X . Cz) = 0 

y . i=1 yi y. yi - 



X (x) so 

"X — 






2 0 



■x 



u • sO 

y - 



(14) 



Although the gradient projection algorithm can be altered to include 
equality constraints [15], they may also be written as two inequalities, 

i .e . 



6 The gradient projection algorithm programmed by D. E. Kirk was 
used in this and the following two applications. 
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( 15 ) 



o . s [ - 7 x f(x,x) + 2. M xi 7 X X x . (x) 3 ^ 0 + 

0- £ L 7^ f(x,Y) + L. M yi 7 Z X yi (y) 3 ^ o + 

The minimum value of f will be zero and the values of x, ju^, 
yielding this minimum will satisfy the Kuhn-Tucker necessary conditions. 

B. DIRECT MINIMIZATION AND MAXIMIZATION BY GRADIENT 
PROJECTION 

1 . Gradient Projection Solution of a Sequence of Constrained 
Optimization Problems 

Since the necessary conditions in Rosen's algorithm are equi- 
valent to the Kuhn-Tucker conditions (see Appendix A), this method may 
be described in terms of two optimization problems . An arbitrary say 
Y_o , was chosen and the minimization problem 

min X €X f - ,Z °) ( 16 ) 

subject to 

X (x) ^ 0. , 

X 

was solved for Xj. Then a solution was found for the maximization 
problem 

ma \eY f ^i '2) ( 17 ) 

subject to 

Xy(x) s 0 

This procedure was continued until some stopping criterion was met. A 
convenient stopping criterion is 

t f (£i/Xi) " f (x.i-i /Xi-i)3 2 < * 
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If this criterion is met, the point (x 4 ,Xi) found is the desired solution 
by the definition of a saddle point. — equation (4) . There remain to be 
shown the conditions under which the sequence of points (x x ,x 2 , . . .) 
and tiei-Xi • • • •) converge to x* and . The Principle of Contraction 
Mapping [13] was applied to the class of functions considered in this 
thesis and conditions were found which ensure convergence for these 
functions (see Appendix B) . 

2 . Simultane ous Minimaximization by Gradient Projection 



not seem to work in general; it is included primarily to point out some of 
the pitfalls of using two dimensions to visualize n-dimensional concepts. 



Before describing this method it should be stated that it does 



The problem of finding a saddle point of f(x,y) subject to 




z = 



X (z) 

— z 



X (y) 
"Y 



Define the "gradient" of f(z) as 



£(z) 



-7 f (x,y) 



(18) 



It was then intended to solve the Maximum Problem 



max f (z) 



z 



subject to 



X (z) > 0 
— z ~ 



with the "gradient" as defined in equation (18), for 
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( 19 ) 




For the case where x and y_ are scalars, the progress of the algorithm may 
be followed on a contour plot and converges quite rapidly. Unfortunately 
this method does not seem to extend to n-dimensions except for isolated 
examples. 
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IV . T WO-D IMENSIONAL EXAM PLES 
The methods described in the preceding section were initially- 
tested on the following two-dimensional example. Find the saddle 
point of 

f (x, y) = x 3 -■ y 3 + xy + 2x + 4y - 6 (20) 

subject to 

M" ^ x < M + 
x x 

( 21 ) 

M“ ^ y <M + 

y y 

The function f is convex in x and concave in y, hence, by Theorems 1 
and 2 a saddle point exists. The region of x,y space for which the 
inequalities (21) are satisfied is called the feasi ble region . If this 
feasible region is large enough, the saddle point will be interior and 
located at 

x* = -1.6 
y* = 1.2 

Clearly, the selection of M + , M" , M + , and M _ in (21) determines 

x x y y 

whether the solution will be interior or constrained. 

From equations (5) through (12) necessary conditions for (x*,y*) 
to be a saddle point are 
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-2x* -2 - y* + n. 



0 



xl 



- H 



x2 



— 2 y* -4 + x* 



x* + M~ SO 
x 



(X* + M- ) ^ = 



(-X* + M* x ) M x2 = 



+ ('yl M y2 
(y* + M - ) fj, 



yi 



(-y* + M+ y ) My 2 



M xl * 



0 

0 

0 

0 

0 

0 



-x* + M + so 
x 




s 0 



A. 



y* 


+ M" 
y 


s o 


M yi 


s o 


y* 


+ M + 

y 


s o 


^y2 


s o 



SOLUTION OF NONLINEAR SIMULTANEOUS EQUATIONS 



( 22 ) 



(23) 



All three methods converged for the several cases of the previous 
example which were examined. 



With M = M + 


= M~ = 


M + = M“ 


= 3. , the three methods 


X 


X 


y y 




converged from the 


starting point (1,-1, 1,1 


,1,1) to 




X* 


= -1.6000 






y* 


= 1.2000 




M xl' M x2' 


M yi' M y2 


= 0.0 


♦ 



The Newton-Raphson method required four iterations, Brown's algorithm 
took three iterations , and the gradient projection root-finding method 
required five iterations. These results are shown in Figs. 1,2, and 3 
respectively . 
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Fig. 1 

Contours of f(x,y) = X s - y 2 + xy + 2x + 4y - 6 showing 
progress of the Newton-Raphson method to an interior saddle 
point . 
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Fig . 2 



Contours of f(x,y) = x a - y 3 + xy + 2x + 4y - 6 showing progress 
Brown's Algorithm to an interior saddle point. 



23 




gradient projection root-finding method to an interior saddle point. 
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